Existing approaches can be categorized into two types. The first type involves leveraging labeled source data for target-domain model selection [9,14-16]. The second type designs unsupervised metrics based on priors of the learned target-domain structure and utilizes the metrics for model selection[17,19,18,20].

However, for the recommender system experiment, there are no natural representations for the candidate models. IS-g/DR-g Off-policy evaluation (OPE) methods can provide an estimate of the accumulative metric. The resulting methods aredenoted asIS-EI andDR-EIrespectively. Asthere arelimited information tobegained byrepeatedly deploying thesame model online, we exclude the models that have been deployed when choosing the next model to deploy for all the methodsincludingAOE. We simulate the "online" deployment scenario as follows: a multi-class classifier is given a set of inputs; for each input, the classifier returns a prediction of the label and only a binary immediate feedback about whether the predicted class is correct is available. They-axisshowsthe gap in the accumulativemetric between the optimal model and the estimated best model by each method.

Model selection is a pivotal process in the quantitative sciences, where researchers must navigate between numerous candidate models of varying complexity. Traditional information criteria, such as the corrected Akaike Information Criterion (AICc), Bayesian Information Criterion (BIC), and Mallows's $\mathit{C_p}$, are valuable tools for identifying optimal models. However, the exponential increase in candidate models with each additional model parameter renders the evaluation of these criteria for all models -- a strategy known as exhaustive, or brute-force, searches -- computationally prohibitive. Consequently, heuristic approaches like stepwise regression are commonly employed, albeit without guarantees of finding the globally-optimal model. In this study, we challenge the prevailing notion that non-monotonicity in information criteria precludes bounds on the search space. We introduce a simple but novel bound that enables the development of branch-and-bound algorithms tailored for these non-monotonic functions. We demonstrate that our approach guarantees identification of the optimal model(s) across diverse model classes, sizes, and applications, often with orders of magnitude computational speedups. For instance, in one previously-published model selection task involving $2^{32}$ (approximately 4 billion) candidate models, our method achieves a computational speedup exceeding 6,000. These findings have broad implications for the scalability and effectiveness of model selection in complex scientific domains.